3-Page Proposal

• Due: Next Tuesday before class

• Aim: Brief description of your project that follows final write-up structure

• Submission: To OSF (add me as collaborator) and post link on Brightspace

Proposal structure

Reviewing Literature

• What do we know about topic related to \(X\)?

• What do scholars think causes \(X\)?

• What do scholars think \(X\) causes (what are its effects)?

• Which particular cases do they study?

Relying on Reputation

• Books from a university press with a reputation

  • Cambridge, Princeton, MIT, Harvard, Cornell, Oxford, Stanford, Michigan
  • Beware of mimicry by bogus presses (Cambridge Scholars Publishing - bad, Cambridge University Press - good)

• Journals with a reputation for publishing good research

  • American Political Science Review, American Journal of Political Science, Journal of Politics, British Journal of Political Science, Political Science Research and Methods, Quarterly Journal of Political Science, Political Analysis
  • Subfields: International Organization, Journal of Conflict Resolution, International Studies Quarterly, Journal of Peace Research, World Politics, Comparative Political Studies
  • In Economics: American Economic Review, Quarterly Journal of Economics, Econometrica, Journal of Political Economy
  • Especially helpful: Annual Review of Political Science, Journal of Economic Literature

Reading Yourself

Let’s Look over the Example

Open Science Framework

• Resource for storing research information

• Each project is stored in a repository with version control

• Main storage for pre-analysis plans for projects

• Let’s go and create a project: osf.io

Statistical hypothesis testing

• Statistical hypothesis testing is a thought experiment

• What would the world look like if we knew the truth?

  • Average treatment effect is \(0\)
  • Each individual effect is \(0\)
  • Sample mean is equal to \(X\)
  • etc.

Conducted with several steps:

Null and alternative hypothesis

• Null hypothesis: Some statement about the population parameters.

  • The “devil’s advocate” hypothesis \(\Rightarrow\) assumes what you seek to prove wrong
  • Ex: Biden’s approval is the same as election result
  • Ex: Treatment effect is zero for everyone
  • Denoted \(H_0\)

Practicum example

• Parameter: Average Treatment Effect (ATE) \(\mu_T − \mu_C\) of encouragement on support for government policies

  • \(\mu_T\): Average support for government policies if everyone received encouragement
  • \(\mu_T\): Average support for government policies if no one received encouragement

\(P\)-value

• \(p\)-value (based on a two-sided test): Probability of getting an (absolute) difference in means this big (or bigger) if the null hypothesis were true

  • Lower \(p\)-values \(\Rightarrow\) stronger evidence against the null

• Intuition: How likely are we to observe what we observe if the null hypothesis is true?

Observed Practicum Data

Practicum Data under Sharp Null

How to calculate?

• We can use the formula under CLT as we did in the practicum

  • note that null here is slightly different, but it is not consequential for most applications

• Or we can do randomization inference:

  • Assume our sample is population and redraw treatment assignment many times calculating difference-in-means each time

Z <- c(1, 1, 1, 0, 0, 
       0, 0, 0, 1, 1)

Y <- c(0.859, 1.930, 0.875, 2.944, -1.015, 
       -0.064, 1.624, -0.411, 1.048, -0.282)

dim_observed <- mean(Y[Z == 1]) - mean(Y[Z == 0])

se <- 
  sqrt(
    var(Y[Z == 1])/length(Y[Z == 1]) + 
      var(Y[Z == 0])/length(Y[Z == 0])
  )

2 * pnorm(abs(dim_observed), sd = se, lower.tail = FALSE)

[1] 0.7382426

Z <- c(1, 1, 1, 0, 0, 
       0, 0, 0, 1, 1)

Y <- c(0.859, 1.930, 0.875, 2.944, -1.015, 
       -0.064, 1.624, -0.411, 1.048, -0.282)

dim_observed <- mean(Y[Z == 1]) - mean(Y[Z == 0])

dim_simulated <- 
  replicate(1000, {
    Z_simulated <- sample(Z)
    mean(Y[Z_simulated == 1]) - mean(Y[Z_simulated == 0])
  })

mean(abs(dim_simulated) >= abs(dim_observed))

[1] 0.739

Testing errors

• A \(p\)-value of \(0.05\) says that data this extreme would only happen in 5% of repeated samples if the null were true.

  • In other words our test results are not always correct!
  • 5% of the time we’ll reject the null when it is actually true.

College sports 🏈 and elections 🗳️

• Irrelevant events affect voters’ evaluations of government performance by Healy, Malhotra, and Mo (2010)

• Summary:

  • Previous research finds that natural disasters and other unpredictable events have an effect on incumbent politicians’ support
  • Authors argue that even events completely unrelated to politics can have an effect on incumbent’s support
  • Consider case of college footbal and basketball games
  • Mix of observational and survey experimental evidence
  • Several robustness excercises

Digging into argument

• Should we expect events unrelated to politics to have an effect on voting?

  • Do government investments and response matter?
  • Do psychological factros matter?
  • What else could matter?

Study 1: College Football and Elections

• What are the dependent and independent variables?

• How are those operationalized?

Controls

• What does it mean to control for the factors (covariates) in estimation? Why would we do that?

Robustness and Placebo tests

• What is robustness test?

Evidence

Study 2: Survey experiment

• What are the dependent and independent variables?

• How are those operationalized?

Evidence

Story of false-positives 😱

• College football, elections, and false-positive results in observational research by Fowler and Montagnes (2015)

How do they show this

• What is dependent and independent variables? How are they operationalized?

• What tests do they run? Why do they run the same regression initially?

How do they explain these results

• Fowler and Montagnes (2015) acknowledge that Healy, Malhotra, and Mo (2010) also ran robustness/placebo tests, BUT:

Broader implications for research design

• Think about all possible implications of theory and see if you can test them with your or other available data

  • This is especially important for observational studies

• Operationalization of variables is crucial!

• If you run many tests of the same hypothesis you need to adjust for multiple comparisons since you can come up with significant results just by chance